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EXTRACT TEXT BOXES; APPLY TEXT 
RECOGNITION; GENERATE ROLENAME 
VS ACTOR-NAME LIST 



-100 



EXTRACT FACE BOXES; APPLY FACE 
RECOGNITION; (EIGENFACES, MODEL 
BASED, ETC.) GENERATE FACE MODELS 



-105 



IF CLOSED CAPTIONING NOT 
AVAILABLE PERFORM AUDIO 
SEGMENTATION AND SPEECH 
TO TEXT CONVERSION 
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INITIALIZE i T0 1 
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-115 



NAMENi IN THE LIST ROLE-NAME- 
ACTOR-NAME LIST DO THE FOLLOWING 



SEARCH FOR ROLE-NAME 
IN THE TRANSCRIPT 



120 



USING THE TIME CODE, SEARCH THE 
CORRESPONDING VIDEO FRAMES IN 
THE SAME OR FOLLOWING SHOTS 
FOR FACE MODELS 
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ASSIGN EQUAL CERTAINTY THAT THE 
NAME IS ASSOCIATED WITH ONE OF 
THE FOUND FACES. 



-130 



FROM 
STEP 160 
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FIG. 1A 



FROM 
STEP 145 
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TO 
STEP 115 
A 



FROM 
STEP 130 



1 



ELIMINATE FACES WHICH 
PRONOUNCE THE NAME (USE LIP 
SYNCING METHOD) 



TO 
STEP 115 
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SCAN THROUGH THE REST OF THE 
TRANSCRIPT AND UPDATE CERTAINTIES 
FOR EACH NEXT APPEARANCE OF ROLE-NAME 
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i=i+1 



IF THE CERTAINTY FOR A FACE MODEL 
AND ROLE-NAME ASSOCIATION IS 
ABOVE A THRESHOLD 



YES 



GOBACKANDBACKPROPAGEBY 
MARKING ALL THE SEGMENTS WHERE 
THIS FACE APPEARED 



STORE ROLE-NAME AND ACTOR NAME 
AND FACE MODELS IN A DATABASE 
FOR FUTURE RETRIEVAL 



NO 



IF Ni IS THE LAST ELEMENT 
OF THE ROLE-NAME LIST 



YES 



NO 
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150 



-155 



160 



SORT THE LIST OF FACE MODELS AND 
KEEP THE n MOST PROMISING FACE 
MODELS FOR FUTURE USE IN THE DATABASE 
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FIG. 1 B 
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ROLE QUESTIONS: WHO ACTS AS IN THE 
ROLE OF MR/MS. X? WHEN DID MR. X 
APPEAR? SUMMARIZE THE MOVIE 
FROM MR. X'S POINT OF VIEW 



ACTOR QUESTIONS: WHO IS MR/MS. Y 

IN THE MOVIE (WHICH ROLE)? 
WHAT DID HE SAY? SUMMARIZE THE 
MOVIE FROM MR. Y'S POINT OF VIEW 



215 



V 

220 



DATABASE: FOR EACH MOVIE: 
ROLE NAME WITH ACTOR NAME LIST, FOR 
EACH ROLE NAME: VIDEO SEGMENTS 
(FROM TO) THAT IT APPEARS IN, FOR 
EACH SEGMENT, THE SENTENCES 
SPOKEN BY THE ROLE 



210 



225 



DATABASE OF DIRECTORS/MOVIES/ACTORS 



FIG. 2 
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OUTPUT:NAME, OR SENTENCE 
OR VIDEO SEGMENT, OR A LIST 
OF VIDEO SEGMENTS 



USER 
DISPLAY 



335 



315 



SEARCH ENGINE 
MODULE 



530 



LIP READING 
MODULE 



305 



i 
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PROCESSOR 



RAM 



320 
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STORAGE 
DATABASE 



DETECTION 
MODULE 



EXTRACTION 
MODULE 



i 
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VIDEO 
PROGRAM 
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